1. The Data¶
On dispose des données de vente journalière de chez BND. L’extraction contient l’historique de ventes entre le premier trimestre 2016 jusqu’à la 3 semaine de janvier 2018.
Les ventes sont groupées au niveau P2 où tous les produits sont pris en compte séparément et en incluant les ventes promotionnelles. Le groupement des ventes par client est fait au niveau C4 où les résultats sont sommés par distributeur.
Data Preparation¶
product_raw_df.head()
Week Resampling¶
La fluctuation des ventes journalières de certains produits étant très importantes, nous allons considérer les ventes sur la base des historiques hebdomadaires.
weekly_product_df = product_df.resample('W',axis=1).mean()
product_df.head()
product_df.head()
Tailing Zeros: No longer sold¶
Remove the products that werent sold in the last 15 weeks
t = 15
Leading Zeros: Recently launched¶
Remove the products which werent sold in the 15 first weeks
t = 15
product_df_clean = product_df
print(product_df.shape)
product_df.head()
Rolling Average¶
Apply a rolling average filter using a window of size = 4 , then remove rare sales
window = 4
Remove Outliers - Winsorizing¶
Winsorizing is to set all outliers to a specified percentile of the data; for example, a 90% winsorization would see all data below the 5th percentile set to the 5th percentile, and data above the 95th percentile set to the 95th percentile
t = 4
Mainly Zeros: Rare sales¶
Remove the products that werent sold for at least 5 weeks
t = 5
Normalize Data¶
We end up with 2846 product/client . Apply a Z-normalization (subtract the mean and divide by the standard deviation) in order to get a scale-independent data
product_df_full.head()
hlp.Clusters_plot(X= X_pca, labels = np.zeros(len(X_pca)))
PCA 01¶
Affiche une tendance croissante dans le temps.
plt.plot(range(nb_col), princ_axis [1,:])
PCA 02¶
Oppose les ventes en terme de saisonnalité. Les produits corrélés avec cet axe ont tendance à être vendu durant les périodes d’Automne ou début d’été.
plt.plot(range(nb_col), princ_axis [2,:])
corrSamples = hlp.GetMostCorrelatedTo(X_pca,component,index=product_df.index)
3. Modeling - Clustering Algorithms¶
Try out Hierarchical clustering, kMeans and kMedodis on raw (cleaned) data. Then, do a PCA plot to visualize the result of the clustering on the principal components
product_df_full.head()
Métriques d'évaluation¶
SSE: La somme des carrés est une mesure de variation ou d'écart par rapport à la moyenne. Elle représente la somme des carrés des différences par rapport au centre du cluster.
Silhouette Score: Permet d'évaluer si chaque point appartient au « bon » cluster : est-il proche des points du cluster auquel il appartient ? Est-il loin des autres points ?
def getSSE(samples,labels):
return np.sum( (samples-labels)**2)
def getSilouhaite(samples,labels):
return metrics.silhouette_score(samples,labels)
Agglomerative - (CAH)¶
Le principe est de rassembler les individus qui affichent des courbes similaires (en distance euclidienne ou en corrélation). Les distances sont calculées entre les produits deux à deux. Plus deux observations seront dissemblables, plus la distance sera importante. La CAH va ensuite rassembler les individus de manière itérative afin de produire un dendrogramme ou arbre de classification
hierarchy.linkage(X_z, method='complete',metric='euclidean')
hierarchy.dendrogram(Z, truncate_mode='lastp', p=100, leaf_rotation=90., leaf_font_size=7., show_contracted=True)
plot_inertie()
Critère de Ward¶
La méthode de Ward consiste à regrouper les classes de façon que l'augmentation de l'inertie interclasse soit maximum.
n_cluster = 116
ward = AgglomerativeClustering(n_clusters=n_cluster, linkage='ward').fit(X_z)
BIRCH Algorithm¶
Used to perform hierarchical clustering over particularly large data-sets.The advantage of using BIRCH algorithm is its ability to incrementally & dynamically cluster incoming.
labels_birch = Birch(n_clusters= n_cluster, threshold=0.5, compute_labels=True).fit_predict(X_z)
Partitionning Algorithms¶
SOM (Cartes topologiques auto-organisatrices)¶
Les cartes auto-organisatrices sont constituées d'une grille. Dans chaque nœud de la grille se trouve un « neurone ». Chaque neurone est lié à un vecteur référent, responsable d'une zone dans l'espace des données (Un certain nombre de courbes similaires).
som = sompy.SOMFactory().build(X_z, mapsize, mask=None, mapshape='rectangular', lattice='rect', normalization='var', initialization='pca', neighborhood='gaussian', training='batch', name='sompy')
K-Means Algorithm¶
The main idea is to define k centers, one for each cluster. then take each point belonging to a given data set and associate it to the nearest center. When no point is pending, re-calculate k new centroids as barycenter of the clusters. A new binding has to be done between the same data set points and the nearest new center. repeat until no more changes are done ie centers do not move any more.
K-means: Validate the number of clusters¶
plot_inertia()
plot_silhouaite()
Apply K-means model¶
kmeans = KMeans(n_clusters=92).fit(X_z)
K-medoids (PAM Algorithm)¶
The k-medoids algorithm is a clustering algorithm related to the k-means algorithm. K-means attempts to minimize the total squared error, while k-medoids minimizes the sum of dissimilarities between points labeled to be in a cluster and a point designated as the center of that cluster. In contrast to the k-means algorithm, k-medoids chooses datapoints as centers ( medoids or exemplars).
K-medoids Validate the number of clusters¶
Custom distances calculation¶
corr_distance = squareform(pdist(X_z, 'correlation')) #Pearsons
euclid_distance = squareform(pdist(X_z, 'euclidean'))
plot_inertia()
plot_silhouaite()
label, medoids_euc = kMedoids.cluster(euclid_distance,k= 113)
df.nlargest(7,"SSE")
Display Clustering results¶
disp = hlp.Cluster_series_plot(data_df = product_df_full, cluster_df = eucl_df,headers = row_headers)
Forecast quality on clusters - First tests¶
In this section we'll build forecasts on a cluster level using classical Holt-Winter's models. Our goal is to evaluate the quality of the prediction on individual series composing the each cluster after performing a weighted split.
The evaluation metric used is the MASE (mean absolute scaled error) which is a scale-independant metric calculated wrt the MAE
group_series.head()
Building Holt-Winter's Models¶
forecast = pd.DataFrame(forecasts,columns = group_series.columns[-fc:],index = group_series.index)
Forecast Split Forecast¶
display(mase_variation.describe())
More numbers...¶
The percentage of clusters where the forecast quality increases at the split level: We compare the average forecast quality at the split level to the forecast on the aggregation level of the cluster.
The percentage of "good" forecasts where the quality decreases with less than 20%
The percentage of products where the split forecast is better than a specific forecast
display(mase_variation[good_splits].sort_values(by=["Agg"]).head())
Displaying forecasts¶
cluster = 99
plt.show()
Next Steps¶
- Improve data preparation process (try different window size, winsorize params)
- Fine tune the clustering model with an automated model selection (random init, number of clusters)
- Improve the split mode